DocTags parser in docling-core + VLM corpus-conformance harness (#152, #153) - #154
Merged
Merged
Conversation
The follow-up docling-project#77 left open: measure docling.rs's --pipeline vlm against Python docling's VlmPipeline over the PDF corpus. scripts/conformance/vlm_conformance.sh — drives both sides against the SAME OpenAI-compatible endpoint (the granite shim; llama.cpp-family servers strip the DocTags structure tokens, see docling-project#77) so model and server are held constant and diffs isolate rendering/parsing/assembly. Reports per-fixture whitespace-normalized character similarity + byte-exactness and a corpus mean; outputs cached under target/vlm-conformance/ for triage (delete to regenerate — each side costs minutes of GPU per fixture). A measurement, not a gate: always exits 0 with numbers for PDF_CONFORMANCE.md. FIXTURES env restricts the corpus subset. scripts/conformance/vlm_convert.py — the reference side: docling's VlmPipeline via ApiVlmOptions (DOCTAGS response format, temperature 0, import-path compatible across docling minor versions). granite_vlm_server.py grows corpus-run ergonomics: a persistent TRITON_CACHE_DIR (kernel recompilation dominated per-page latency in the docling-project#77 bring-up) and --warmup to pay first-request compilation before the corpus starts. docs/PDF_CONFORMANCE.md: "VLM pipeline conformance" section marking the measurement pending and naming the known render-scale asymmetry so drift triage starts from the right bucket. Refs docling-project#153 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…cling-project#152) The DocTags→DoclingDocument parser docling-project#152 asks about, as a docling-core module (mirroring Python docling-core, which hosts DocTagsDocument): a tolerant lexer + recursive-descent walk over the token stream the docling VLMs emit. DocTags is not XML — loc tokens, OTSL cell markers, picture classes and checkboxes are unclosed, captions nest inside <otsl>, and generation truncates mid-element — so the grammar is parsed natively instead of being lexically rewritten into DocLang (docling-project#77's interim bridge, now deleted): - <loc_*> runs become real layout provenance: Node::Located wrappers and per-node locations (table/list/formula), which the lexical bridge threw away; - OTSL tables keep their span/header structure (TableStructure: ched/rhed headers, lcel/ucel/xcel continuations), not just the text grid, with the in-otsl caption emitted as a paragraph before the table (docling reading order); - section_header_level_K → heading K+1, title → 1 (docling Markdown parity, pinned against the pg9 groundtruth in docling-project#77); - page_header/page_footer → Node::PageFurniture (Markdown/JSON omit); - code recovers its <_lang_> token as the language; formulas map to Node::Formula; - everything unrecognized degrades: unknown tokens are transparent, stray block-level text becomes paragraphs (the token-stripping-server shape from the docling-project#77 bring-up), truncated elements stop at the next block start instead of swallowing the page. parse() never fails. API: doctags::parse(&str) / parse_pages(iter) — pure Rust, no new dependencies, panic-free, wasm32-safe (checked). docling/src/vlm.rs: fragments now route by grammar — DocTags-shaped answers (loc/otsl/section_header markers) go to the new parser, DocLang XML keeps the existing reader path, untagged prose keeps the line-per-paragraph fallback. The interim DocTags→DocLang lexical translator and its tests are gone; unit tests cover the routing + wrapper stripping, docling-core tests pin the full markup semantics (real granite shapes: spans, truncation, degraded loc+text, entities), and a new mock-server e2e exercises the DocTags route end to end. Refs docling-project#152, docling-project#77 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
.doctags input format (docling-project#152) Rounds out the DocTags parser to the full vocabulary Python docling-core's DocTagsDocument covers: - <checkbox_selected>/<checkbox_unselected> → Node::CheckboxItem (Markdown task-list rendering), both as block-level tokens and inside a <text> wrapper; - <key_value_region> (DocTags) / <field_region> (DocLang naming) → Node::FieldRegion, tolerant over numbered DocTags pairs (<key_1>…<value_1>…) and the DocLang <field_item> shape, including <unmatched_value>; - .doctags/.dt files are now a first-class InputFormat (DocTags) wired through the converter to docling_core::doctags::parse — a saved model transcript converts like any other document; name maps added to the Python/Node bindings' format tables. vlm_conformance.sh from the first live corpus run: per-side progress lines (a multi-page fixture takes minutes per page and looked hung), and cached outputs write through a temp file + rename so Ctrl+C never caches a half-written result. README notes the DocTags input; the docling-project#152 scope — parser in docling-core, model-typed API, tolerance, geometry — is now fully implemented. Refs docling-project#152 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two follow-ups to the VLM pipeline (#77), one branch because the second builds on the first.
DocTags parser in docling-core (#152)
docling_core::doctags — a native parser for the token markup docling's VLMs (granite-docling / SmolDocling) emit, replacing #77's interim lexical DocTags→DocLang bridge (now deleted from vlm.rs). DocTags is not XML: <loc_N> runs, OTSL cell markers, picture classes and checkboxes are unclosed tokens, captions nest inside , and generation truncates mid-element — so this is a tolerant lexer + recursive-descent walk that never fails: everything unrecognized degrades to text or is skipped.
What the parser keeps that the bridge threw away:
layout provenance — loc_* runs become Node::Located wrappers and per-node locations on tables/lists/formulas;
table structure — TableStructure with ched/rhed headers and lcel/ucel/xcel span continuations, not just the text grid; the in- caption becomes a paragraph before the table (docling reading order);
code language from the <lang> token; formulas as Node::Formula; page furniture as Node::PageFurniture (omitted from Markdown/JSON like the ML pipeline's furniture).
docling-parity choices pinned by tests against real granite-docling output from the #77 bring-up: section_header_level_K → heading K+1 (# is the title), matching the pg9 groundtruth; degraded token-stripped output (loc runs + bare text) survives as paragraphs; truncated elements stop at the next block start instead of swallowing the page.
API: doctags::parse(&str) / parse_pages(iter) — pure Rust, zero new dependencies, panic-free, wasm32-safe (checked with the CI wasm target command). vlm.rs now routes fragments by grammar: DocTags → this parser, DocLang XML → the existing reader, untagged prose → line-per-paragraph fallback.
Answers #152's three questions in code: in scope — yes; in docling-core — yes, as a module next to doclang; API mirrors Python docling-core's DocTagsDocument. Contributions on top (key-value regions, checkboxes, inline styling) welcome.
Also: checkboxes, key-value regions, and .doctags/.dt as a first-class input format wired through the converter and bindings — the module now covers the full vocabulary of Python docling-core's DocTagsDocument.»
VLM corpus-comparison harness (#153)
scripts/conformance/vlm_conformance.sh — runs the PDF corpus through docling.rs (--pipeline vlm) and Python docling's VlmPipeline, both against the same endpoint (scripts/dev/granite_vlm_server.py; llama.cpp-family servers strip the DocTags structure tokens), and reports per-fixture whitespace-normalized character similarity + byte-exactness + a corpus mean. Outputs cached under target/vlm-conformance/ for triage; FIXTURES= restricts the subset. A measurement, not a gate — always exits 0.
scripts/conformance/vlm_convert.py — the reference side via ApiVlmOptions (DOCTAGS response format, temperature 0), import-compatible across docling minor versions.
shim ergonomics for corpus runs: persistent TRITON_CACHE_DIR + --warmup (kernel compilation dominated per-page latency during bring-up).
docs/PDF_CONFORMANCE.md: "VLM pipeline conformance" section — measurement pending, with the known render-scale asymmetry named so drift triage starts in the right bucket.
The actual corpus run needs a GPU (see #153); numbers land in PDF_CONFORMANCE.md once it happens.
Testing
Unit + e2e green: docling-core DocTags tests (real granite shapes — spans, truncation, degraded output, entities), vlm.rs routing tests, mock-server e2e including a DocTags-path leg; full docling-core/docling suites, clippy clean, cargo fmt, wasm32 check. Live-verified against granite-docling on an RTX 3080 through the transformers shim (structure tags intact → ## headings, hoisted caption, span-correct table).
Refs #152, #153 (follow-up to #77)